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NOTE: Content analysis is a set of methods for message analysis that enjoys a long history, dating to 
the early 20th century (Berelson, 1952; Neuendorf, 2002; Smith, 2000), and is one of the fastest- 
growing methods in the social and behavioral sciences, while also having extended its reach into 
business, the sciences, and other disciplines (Neuendorf, 2017). Content analysis, in its most common, 
quantitative form, may be defined as a “summarizing, quantitative analysis of messages that follows the 
standards of the scientific method (including attention to objectivity/intersubjectivity, a priori design, 
reliability, validity, generalizability, replicability, and hypothesis testing based on theory) and is not 
limited as to the types of variables that may be measured or the context in which the messages are 


created or presented” (Neuendorf, 2017, p. 17). 
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INTRODUCTION 


In a recent study, Geoffrey Supran and Naomi Oreskes (hereafter “S&O”; 2017) claim to have 
found (1) “a discrepancy between what ExxonMobil’s scientists and executives discussed about 
climate change privately and in academic circles and what it presented to the general public” and (2) 
“that ExxonMobil’s AGW [climate change] communications were misleading” (p. 15). In the course 
of their study, S&O cite my methods textbook (Neuendorf, 2002) as a source for their content 
analysis methods. After a detailed review of the study, its supplementary information (‘SI’), and the 
documents S&O analyzed for their study, I have concluded that S&O’s content analysis does not 
support the study’s conclusions because of a variety of fundamental errors in their analysis. S&O’s 
content analysis lacks reliability, validity, objectivity, generalizability, and replicability. These basic 
standards of scientific inquiry are vital for a proper content analysis, but they are not satisfied by the 


S&O study. 


As described in greater detail in this report, seven fundamental flaws undermine the S&O research 


and are fatal to its principal findings: 


I. Non-representative, confounded sampling. S&O have selected a non-representative sample of 
climate communications. The selection process was not objective or consistent across document 
types. The authors have also improperly grouped together communications that vary across time and 
by author and audience. Of even greater concern, the selection process groups together statements 
as though they were issued by a single corporate entity during times when Exxon Corporation and 
Mobil Oil Corporation were separate companies and misleadingly presents them as though they 
were issued by a unitary entity throughout the time period. As a result, S&O improperly classify 
statements from two separate companies (Exxon and Mobil) as though they were issued from one 


company. 


II. Inappropriate coders. To maintain objectivity, content analysis coding ought to be conducted 
by coders who are at arm’s-length with regard to the research. S&O’s selection of themselves as 
coders is inappropriate because they are not blind to the purpose of the research or independent of 
each other. In fact, they were as non-blind as one could imagine. Moreover, their prior statements 
about climate change and Exxon Mobil Corporation (including Oreskes’ (2015b) tweet, “Did Exxon 


deliberately mislead the public on climate change? Hello. Of course they did!”’) reveal biases against 
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ExxonMobil. The use of highly involved, heavily interrelated, non-blind coders renders the study 


non-replicable. 


III. Flawed coding scheme. As described in the S&O article and the SI, the coding scheme 
deviates from standards of content analysis in a variety of ways. Specifically, the coding scheme 
shows bias against ExxonMobil, is quite complex and difficult for coders other than the co-authors 
to apply, instructs coders to skim articles for material to code, allows context outside the documents 
to guide coding, and calls for resolving coding ambiguities through discussion. None of these are 


appropriate in a content analysis. 


IV. Lack of research questions. The study lacks theory-backed hypotheses or more general 
research questions, undermining any claim to objectivity. The SI purports to create research 
questions, but these prove to be presuppositions based on the co-authors’ assumptions rather than 


questions that are rigorously examined. 


V. Lack of disclosure. S&O fail to disclose their rationale for selecting some of the documents in 
their sample, including the set of advertorials that S&O assert “misled the public” (S&O, 2017, p. 1). 
S&O also omit essential details about their coding scheme for the content analysis. These omissions 


render the study non-replicable. 


VI. Unwarranted inference. S&O improperly infer from content analysis that “ExxonMobil misled 
the public” (S&O, 2017, p. 1). Content analysis cannot legitimately be used to reach conclusions 
about the effect particular statements have on the public. Additional information is needed about 
how those who read the climate change communications in question responded to them, but S&O 
provide no data documenting public reception. Further, any inference to the intentions of 
ExxonMobil personnel who authored the various communications is also unwarranted from the 


content analysis. 


VII. Consensus measurement. S&O note that they rely on a technique known as consensus 
measurement, used in prior work of one of the co-authors. Unlike content analysis, which is well 
established, consensus measurement does not appear to be a general, scientific method, but instead, 
a conclusion regarding consensus about climate change opinions in search of a method. 


Accordingly, the application of consensus measurement lacks reliability, objectivity, and validity. 
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In light of these significant errors and omissions, the conclusions reached by S&O are not sound, 


and should not be relied upon. 


BACKGROUND 


The S&O study, published in Environmental Research Letters (with online SI), examines climate change 
related communications of four different types: peer-reviewed documents, non-peer-reviewed 
documents, internal corporate documents, and advertorials. S&O indicate that they have used 
quantitative content analysis to examine these documents, and have cited my methods textbook 
(Neuendorf, 2002) as a source of information for the methods of content analysis. S&O also claim 
to be undertaking what they call a “challenge” (p. 2) posed in a blog post by ExxonMobil asking the 
public to read a set of documents to determine whether the company suppressed its climate change 
research (Cohen, 2015). S&O alter the nature of the challenge by (1) shifting from the question of 
suppressed research to a different claim that ExxonMobil made misleading communications to the 
public and (2) adding documents to their sample that were not referenced in the original blog post 


(S&O, pp. 2-3). 


Disseminating the results of their study to the general public, the authors describe their conclusions 
in an op-ed piece published in the Los Angeles Times. “The result: a systematic discrepancy between 
what ExxonMobil scientists communicated in their scientific articles and internal reports, and what 
the company told the public in ‘advertorials-—advertisements in The New York Times masquerading 
as editorials. In other words, our study showed that ExxonMobil misled the public about climate 


science and its implications for decades” (Oreskes & Supran, 2017). 


S&O have a documented history of offering opinions about climate change generally and the 
conduct of ExxonMobil specifically. Since claiming nearly 15 years ago that a scientific consensus on 
climate change exists (Oreskes, 2004), Oreskes has equated expressions of disagreement with this 
consensus by scientists and by ExxonMobil to the tobacco industry’s suppression of information 


regarding tobacco’s negative health effects (Oreskes & Conway, 2010; Oreskes, 2015a). 
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Further, on the same day that ExxonMobil published its blog post asking the public to read its 
selection of documents, Oreskes issued the tweet: “Did Exxon deliberately mislead the public on 


climate change? Hello. Of course they did!” (Oreskes, 2015b). 


Also, Supran has supported the fossil fuel divestment movement. For example, well before the S&O 
study was published, Supran, in a tweet, endorsed the view that “Exxon’s actions may have 


imperiled all of humanity. It’s time to divest” (Supran, 2016). 


ANALYSIS 


Each of the seven areas of concern about the S&O methods listed above will be elaborated upon in 


the sections below. 


I. The S&O sample is non-representative, and the four stratified subsamples ignore 
important differences in time period, corporate affiliation, specific authorship, and intended 


audience. 


I. A. The sampling is not representative. The sampling strategy of S&O is stratified. The sample is 
stratified on the characteristic of document type: peer-reviewed documents, non-peer-reviewed 
documents, internal documents, and advertorials. Each of these four types is represented by a 


separate sample. 


In order for each of the four samples to be representative of (and therefore generalizable to) the 
relevant type of document, either all documents of that type need to be collected (constituting a 
census of that group of documents) or the documents need to be collected in a probability sample that is 
statistically a good representation of that type. Neither of these techniques was used by S&O, as 
described in the S&O journal article and SI. 


Referring to the 2015 ExxonMobil blog post (Cohen, 2015) that urges readers to examine both 
internal and research documents produced by ExxonMobil, S&O indicate that they have taken “up 


that challenge by analyzing the materials highlighted by the company, and comparing them with 
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other publicly available ExxonMobil communications on AGW” (S&O, p. 2). Yet S&O have strayed 


from the collections that ExxonMobil provided in that challenge, as indicated in their Table 1: 


a. The peer-reviewed documents (n=72) are mostly from the ExxonMobil list, but three 


documents are added from “other” sources; 


b. The non-peer-reviewed documents (7=47) are also mostly from the ExxonMobil list 


(n=32), but 15 additional documents are from “other” sources; 


c. The internal documents (n=32) are a mixture of documents provided by ExxonMobil as 
linked from its 2015 blog post (v=22), documents collected by InsideChmate News (an 
environmental news organization) (v=12, 9 unique to ICN and 3 overlapping with 


ExxonMobil’s list), and one document from an “other” soutce; 


d. The advertorials (v=36) are all from a collection by PolluterWatch, a Greenpeace project. 


None of these four seem to have been produced by a rigorous, well-documented sampling 
procedure that would result in a representative sample. S&O give some statements indicating their 
strategies, but these are not replicable (i.e., repeatable by others). For example, they indicate that the 
sample of internal documents was “the relevant, publicly available internal documents that have led 
to recent allegations against ExxonMobil” (p. 1, SI). There is no explanation of how “relevant” was 


operationalized in the selection of documents. 


And the advertorials were taken only from The New York Times, a publication that is acknowledged as 
having an “elite” readership (Brown & Waltzer, 2005), from a list by PolluterWatch, a project of 
Greenpeace dedicated to “holding polluters accountable.” This limitation and potential bias of the 


advertorial sample has not been discussed or taken into account by S&O. 


Since a main argument of S&O’s conclusion is that the content differed across the types of 
documents and that this indicates some type of biased communication on the part of ExxonMobil, 
then it is important that the four document types were collected in a matched and representative 
fashion. In general, there is no explanation of how documents were collected objectively via 
comparable techniques across document types, and, upon inspection, it is clear that numerous 


documents were hand-picked and not part of a comprehensive sampling strategy. Indeed, S&O 
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admit that there are “countless additional climate change communications from ExxonMobil that 


could be included in future work” (p. 2), thus undercutting the integrity of their own sampling. 


I. B. Further, the separation of the four “type of document” sub-samples is confounded with several other important 
variables. First, there is a confounding of type of document and “#me period in the presentation of 
findings. That is, all pre-1980 documents are internal documents, only one post-1995 document is 
internal, and all post-2004 documents are research pieces. So, when S&O conclude that “there is a 
discrepancy between what different document categories say, and particularly what they emphasize” 
(p. 12), they have not disentangled the variable “document category” (i.e., sample stratum) from time 
period. Thus, it is not really known how much of the difference is attributable to document 
type/category and how much is attributable to time period. While the SI does present figures 
reworked to include only “overlapping publication periods” (1989-2004), all pre-1989 internal 
documents are still included. And, the Fisher’s exact test statistic, which S&O applied to the full set 
of four samples over the 1977—2014, was not repeated for this set of documents from the 


overlapping time periods. 


The S&O study analyzes 187 communications they attribute to ExxonMobil originating over a 38- 
year period. This time span included many changes in society, science, business, and the corporate 
identity and operation of ExxonMobil in particular. In 1999, the separate oil entities Exxon and 
Mobil merged to form ExxonMobil, prompting many changes in corporate structure for the Fortune 


500 oil and gas corporation. 


Thus, in the S&O study, there is also a further confounding of type of document, time period, and 
corporate affiliation (1.e., Exxon vs. Mobil vs. ExxonMobil). S&O wish to compare the content of 
document types, ostensibly from a single corporate source (ExxonMobil). However, S&O 
improperly classify pre-merger documents from two separate corporations (Exxon Corporation and 
Mobil Oil Corporation) as though they were issued from one company. For documents with 


identifiable corporate sources: 


a. Documents in the peer-reviewed sample are essentially all Exxon or ExxonMobil (never 


Mobil), with an even split pre- and post-merger; 
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b. Documents in the non-peer-reviewed sample are nearly all Exxon or ExxonMobil (only 


one originates from Mobil), with far more post-merger than pre-merger (65% vs. 35%); 


c. Documents in the internal documents sample are nearly all from Exxon during the pre- 


merger period; and 


d. The advertorial sample is wholly from Mobil or ExxonMobil (never from Exxon), with far 


mote pre-merger than post-merger documents (69% vs. 31%)". 


So, in comparing the four document types, any analysis is also comparing time period and corporate 
source. For example, comparing advertorials with non-peer-reviewed documents would in essence 
be a comparison of (notably unique) Mobil pre-merger advertorials with ExxonMobil post-merger 
non-peer-reviewed documents. Comparing internal documents with advertorials in the pre-merger 
period would in essence be a comparison of Exxon Corporation documents with Mobil Oil 


Corporation documents. 


Elsewhere, I (Neuendorf, 2002; 2017) propose that when engaging in content analysis, the 
researcher ought to attempt to identify “critical variables,” i.e., those that are vital to a 
comprehensive understanding of (1) the message pool (2) in the specific medium under investigation 
(2017, p. 97). In this case, the examination of documents produced over time by a corporation that 
has experienced significant organizational change over that time demands a look at what critical 
characteristics of the organization may have had a differential impact on the content of the 
documents over time and across types of document. The documents included in the S&O set of 
four samples were authored by a wide range of individuals affiliated with Exxon, with Mobil, and 
with ExxonMobil over a four-decade period. Both corporate affiliation, as noted above, and specific 


authorship seem to be critical variables related to the nature of the four types of documents over time. 


In examining solely the 72 peer-reviewed documents, one finds a mixture of ExxonMobil and 


academic (non-ExxonMobil) co-authors for most documents. The pre-merger peer-reviewed 


' Tn the early portion of the study period (i.e., in the 1980s), Mobil became known for its involvement in the 
development of the advertorial, a newspaper or magazine advertisement that takes the form of a news or editorial 
journalistic piece. The advertorial has become a widely used and integral part of corporate communications across 
industries (Brown & Waltzer, 2006). The unique public relations voice of Mobil, prior to its merger with Exxon, has 
been the focus of scholarly analysis, as noted by S&O (e.g., they cite: Brown & Waltzer, 2005; Crable & Vibbert, 1983; 
St. John, 2014a; 2014b; see also Murphree & Aucoin, 2010; Smith & Heath, 1990). 
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documents are all authored or co-authored by Exxon personnel (i.e., none are by Mobil). Prior to 
the merger, there are 11 Exxon authors noted, with 23 of the 30 pieces including Haroon S. Kheshgi 


as an author. 


In the post-merger period, the collection of peer-reviewed documents becomes more diverse in 
terms of authorship. Obviously, the documents produced after the 1999 merger are all authored or 
co-authored by ExxonMobil personnel, but various divisions of the Exxon Mobil Corporation now 
begin to be represented. While nearly all the Exxon Corporation authors prior to the merger are 
credited as affiliated with the Exxon Research and Engineering Company, after the merger, in 
addition to the newly minted ExxonMobil Research and Engineering Company, six additional 
corporate affiliations appear: ExxonMobil Upstream Research Company, ExxonMobil Development 
Company, ExxonMobil Production Company, ExxonMobil Exploration Company, Exxon Mobil 
Corporation, and ExxonMobil Gas and Power Marketing Company. 


Similarly, the number of engaged Exxon or ExxonMobil authors expands dramatically over time. 
Prior to 1999, 11 different Exxon authors are identified. After 1999, there are 27 different 
ExxonMobil personnel identified as authors/co-authors of peer-reviewed documents. Thus, there is 
increasing variability in authorship over time, which might relate to a greater variety of viewpoints 


represented in the peer-reviewed documents post-merger. 


Another critical variable when considering the four different sample types seems to be intended 
audience. The four document samples were aimed at a range of audiences. Authors across the four 
document types include public relations specialists, climate scientists, engineers, and economists. It is 
unlikely that the same content would be produced by these different types of authors for different 
audiences with different purposes in mind. For example, there seems little reason for an advertorial 
aimed at the general public to talk about stranded fossil fuel assets, a complex and speculative 
economic issue. And economic issues are not likely to be addressed in the purely scientific peer- 


reviewed articles. 


I. C. There are additional discrepancies in the samples. As noted above, the S&O journal article and SI do 
not provide full details as to how the four samples were collected. Some discrepancies emerge when 


examining the 187 documents. 
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The “peer-reviewed” and “non-peer-reviewed”’ distinction is not made by ExxonMobil in its 2015 
blog post (Cohen, 2015). The two lists of research reports presented by ExxonMobil are “peer- 
reviewed publications” and “additional publications.” While this distinction is not clear-cut, in that 
multiple “additional publications” are peer-reviewed, no comparison is made between these two 
types by ExxonMobil in its presentation. S&O do make explicit comparisons between peer-reviewed 
and non-peer-reviewed documents, and their allocation of documents into these two bins is not 
always correct. That is, there are some peer-reviewed documents listed in their non-peer-reviewed 
sample, and some non-peer-reviewed documents listed in their peer-reviewed sample. And, there are 
several documents within the non-peer-reviewed collection for which Exxon/ ExxonMobil 


personnel are reviewers or panelists, not authors. 


These and other discrepancies concerning how the four samples were constituted further call into 


question the rigor of the S&O study. 


In sum, the four stratified samples utilized for the S&O content analysis are not clearly 
representative of the four types of documents the researchers wished to analyze, and the four types 
are confounded with time period and corporate affiliation. Further, specific authorship, variety of 
authorship, corporate division affiliation, and intended audience all seem to be critical variables that 
have not been taken into account when comparing the four sample types and examining potential 
trends over time. And, additional discrepancies with regard to the documents call into question the 


validity of the processes of sampling and document collection. 


II. The coders for the S&O content analysis are highly involved in the topic area, biased, 
heavily interrelated, and not blind to the purpose of the research, rendering them 


inappropriate for the task. 


As indicated in my methods text on content analysis, a coding scheme should be created so that 
once trained, coders “from varied backgrounds and orientations will generally agree on its 
application” (Neuendorf, 2002, p. 9). And, “any a priori requirement in coder qualifications may 
limit the validity of a coding scheme, particularly with regard to external validity, as well as limit the 
replicability of the study” (Neuendorf, 2017, p. 157). While some content analyses, including the 
S&O one, may require intensive coder training or even some type of past knowledge or education, 


“coding should still not be dependent on particular individuals, but rather on a certain classification 
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of individuals (e.g., experienced video gamers, Spanish speakers, or individuals with a type of 
medical qualification) so that coders are still interchangeable within this classification” (p. 157). In 
the case of S&O, the coding procedure as presented does seem to be dependent on the particular 


individuals who were the coders, i.e., the investigators themselves. 


Content analysis coding ought to be conducted with coders who are at arm’s-length with regard to 
the research, in order to maximize objectivity. Optimally, coders should be blind to the research 
questions or goals. In the S&O study, the coders were not blind. In fact, they were as non-blind as 
could be imagined. They were the investigators themselves, as well as an affiliated graduate student. 
In this particular case, the problematic nature of informed coders is magnified by the coders’ long- 
time and intensive involvement in the popular communication of climate change. Further, two of 
the coders have publicly demonstrated particular biases that existed before the execution of the 
S&O study (Oreskes, 2004; Oreskes, 2015a, 2015b; Oreskes & Conway, 2010; Supran, 2016). As 
noted in the Background section above, Oreskes has a long history of negative commentary about 
ExxonMobil’s activities, including the tweet: “Did Exxon deliberately mislead the public on climate 
change? Hello. Of course they did!’ (Oreskes, 2015b). And Supran also tweeted: “Exxon’s actions 


may have imperiled all of humanity. It’s time to divest” (Supran, 2016). 


It is also important that at least two content analysis coders are employed, in order to provide 
reliability assessment. The use of two or more coders in reliability assessment assumes that the 
codings of the coders are produced independently of one another. In the case of S&O, the coders 
are interdependent in their broader work, and it seems likely that they would approach the coding 
task with similar orientations, i.e., that they would not achieve the independence that is expected of 


coders. 


The use of highly involved, heavily interrelated, non-blind coders renders the study non-replicable. 
That is, it is likely that blind coders not involved in climate change research or advocacy would not 
be readily trainable on this coding scheme, would not produce such reliable codings, nor would they 


obtain results highly comparable to those presented by S&O. 


III. S&O employ a coding scheme that includes bias, is complex and requires specialized 


expertise beyond the scheme, and instructs coders to engage in activities that are 
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unacceptable for content analysis, including skimming, looking beyond the documents for 


context, and resolving coding ambiguities through discussion. 


II. A. The coding scheme shows bias. In the section of the SI that seems to be the closest thing available 
to a full codebook, this passage is found: “Moreover, some of the analyzed documents demonstrate 
that ExxonMobil’s use of tentative wording to emphasize uncertainty was, at least sometimes, 
intentional” (p. 2). This statement seems to be a conclusion, not an unbiased instruction to coders, 


and is not indicative of an objective coding scheme. 


III. B. The coding scheme seems to be quite complex and difficult to apply. In particular, the coding of the 
scientific articles (peer-reviewed and non-peer-reviewed) clearly requires expertise beyond that 
provided by the coding scheme as presented in the SI. Additional coder training or education is 
necessary, but this training is not specified. It is doubtful that unaffiliated coders blind to the intent 
of the study would do a very good job using it. This study’s execution is dependent on very 
specialized knowledge, and dependent on the background of the particular individuals executing the 
research. Thus, the study’s objectivity and reliability, and therefore also its validity, are called into 


question. 


II. C. The coding instructions deviate from standard, recommended practice. Unless specific processes are 
proscribed in the coding scheme, using context for determining coding decisions is not standard 
practice. In the S&O research, coders seem to be encouraged to use contextual information. Coder 
notes in the SI include the following: “Context is key to this paper.” Given that different coders are 
likely to have different contextual knowledge approaching the coding task, this precludes objectivity 


and reliability of the content analysis. 


Further, coders were allowed to “skim” articles (S&O, p. 3) to locate codable material, that is, to 
identify coding units. This type of acomplete review by coders does not match any currently 


recommended methods for unitizing (identifying codable units) or for coding in content analysis. 


“Coding ambiguities” were resolved through discussion, which is not a recommended procedure for a 
final coding process in content analysis. Although certainly part of the development process for a 
coding scheme, and for training as well, such discussion should not be part of the final analysis. In 


fact, the SI indicates that “through ‘negotiated agreement’ of discrepancies between coders, 
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intercoder agreement was then calculated” (p. 6). So, the difference between S&O’s “intercoder 
reliability” and “intercoder agreement” is pre-negotiated reliability vs. post-negotiated agreement, it 
seems. This is not commonly accepted practice for content analysis. Given that the pre-negotiated 
reliabilities are all acceptable, there seems no reason to engage in the additional, questionable and 


potentially biased process of negotiated agreement. 


The SI presents “secondary codes,” seemingly for the four sets of “document position” variables. It 
is unclear how these more specific codes were used (they have no code numbers assigned). And it is 
unclear whether reliability assessment was conducted on these secondary codes, or only on the 


primary codes. 


These limitations in the coding scheme, and its presentation, prevent the study from being 
replicable. As one of the goals of science, replicability is an important aspect of content analysis. It 
removes the analysis from the realm of being executable only by certain individuals. One basic 
motivation for content analysis is to measure characteristics of messages that might be influential to 
audience members receiving those messages. And one core assumption is that if extreme technical 
expertise is needed to detect certain characteristics of messages, these characteristics are unlikely to 


be discerned by the general public. 
IV. The S&O study lacks guiding research questions, limiting its objectivity. 


Content analysis is optimally conducted within the framework of the scientific method. It is 
therefore important that the research be guided by specific hypotheses derived from broadly 
generalizable theory, or more general research questions optimally dertved from theory or past 
scholarship. In this way, the relative objectivity of the research endeavor is supported. The S&O 


journal article does not present hypotheses or research questions. 


The S&O SI does indicate that “research questions” were created (but not actually presented in the 
article or the SI) “in order to determine whether the corporation misled consumers and/or 
shareholders by making public statements that cast doubt on climate science and its implications, 


and which were at odds with available scientific information and with what the company knew” (p. 
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1). This phrasing is more consistent with presuppositions than queries, revealing assumptions made 


by the researchers, and indicating a bias that precludes the study’s objectivity.’ 


Even mote definitive, a public statement of the study’s major conclusion was made by one of the 
authors prior to the execution of the study. As noted above, Oreskes issued the following statement 
on social media on October 21, 2015: “Did Exxon deliberately mislead the public on climate 
change? Hello. Of course they did!” (Oreskes, 2015b). This pre-study conclusion violates basic 


tenets of scientific research. 


V. S&O fail to disclose important aspects of their document sampling and their coding 


scheme, making the study non-replicable. 


While S&O have provided a good quantity of information in their journal article and SI, they have 
failed to provide essential material for the study to be repeatable by others. The S&O research lacks 
full disclosure of all details regarding sampling and content analysis coding processes, such that even 
if the known flaws were able to be rectified, there are additional possible important limitations to the 


design of the research. 


The coding scheme for the S&O content analysis is (necessarily) only briefly described in the journal 
article, while the SI provides greater detail. Still, the reader cannot be certain of the full scope of the 
coding scheme. There is no standalone codebook with full instructions, as one would expect to have 


been provided to the coders, or an outline or notes regarding coder training. 


Unitizing is not addressed in the documentation provided in the journal article or its SI. While 
ostensibly the unit of analysis is the individual document (e.g., a peer-reviewed research article, or an 
advertorial), the SI presents the full collection of text segments that were coded in particular ways, 
and therefore served as units. As referred to in the SI, the researchers are providing the “coded 
Endorsement (EP), Impact (EP) [sic], and Solvable (EP) [sic] Points . . . and substantiating 
quotations (coding units)” (p. 16). So, the units that were coded were not the full documents, and 


the segmentation of the documents into these coded units has not been explained. No unitizing 


2 An example of a formal research question would be: “RQ1—Do the four types of documents (peer-reviewed, non- 
peer-reviewed, internal, and advertorials) differ in the extent to which they explicitly endorse, with quantified support, 
the view that AGW is real and human-caused?” (This would relate to the S&O measurement called EP1.) 
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reliability was assessed (a process that is recommended but unfortunately not always conducted). 


Without clear instructions as to a unitizing process, such units are likely to fluctuate across coders. 
In sum, these deficiencies in reportage alone render the study non-replicable. 


VI. The S&O study makes an unwarranted inference from its content analysis to its claim as 
to how ExxonMobil’s climate change communications were received. Further, any inference 
to the intentions of ExxonMobil personnel who authored the various communications is 


also unwarranted from the content analysis. 


Content analysis is particularly suited to analyze the “message” component in the classic Source- 
Message-Recetver model of communication. To infer from content analysis findings to source 
characteristics or receiver outcomes, the researcher needs to either (1) collate the content analysis 
data or findings with data from the sources or receivers or (2) have valid information on “well worn 
pathways” of relationships between sources and messages, or between messages and receivers, from 
substantial previous research. The S&O study attempts to draw some inferences to sources and 


receivers, without having done either (1) or (2). 


First, the investigators go beyond the boundaries of what one may conclude from content analysis 
by making assumptions about the source(s) of the messages. S&O contend that ExxonMobil misled 
the public, but the term “misled” implies a potential intention on the part of sources that has not 
been examined in the S&O research. Importantly, the diversity of the ExxonMobil sources 
described in section I.B. above runs counter to the S&O assumption of a unified source acting with 


intentionality. 


In particular, documents that predate the 1999 merger come from completely separate organizations, 
making it impossible to attribute some intention to mislead from any differences found in 
documents from the two corporations. All the peer-reviewed scientific documents pre-merger were 
authored/co-authored by Exxon personnel, while all advertorials were from Mobil staff. It does not 
seem logical to expect that Mobil should have adopted Exxon’s scientific position on climate change 


at that time and presented those views in advertorials in which Exxon’s name did not appear. 


Most of the research documents, peer-reviewed and non-peer-reviewed, were co-authored by a 


mixture of ExxonMobil (or Exxon, or Mobil) employees and academic researchers. It is not 
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appropriate to assume that the academic researchers were presenting an ExxonMobil corporate 
perspective, as their faculty status would be contingent on conducting independent, peer-reviewed 
research. This further negates the notion that the documents presented—or should have presented—a 


unified corporate viewpoint. 


Second, with regard to the receivers of the messages, S&O maintain that those members of the public 
who were exposed were “misled,” without ascertaining how many consumers may have been 
exposed to the messages, and whether they actually changed their beliefs as a result of exposure. 


This unwarranted inference is not supported by the content analysis research. 


VII. S&O rely on consensus measurement, a method that does not appear to qualify as an 


accepted, scientific method. 


VIL A. Consensus measurement is not a standard, widely accepted method. Consensus measurement, in 
comparison to content analysis, is not a standard, time-honored research technique. It does not 
seem to be a methodology or a set of methods that may be applied to a range of phenomena, Le., it 
is specifically the process of quantifying scientific consensus with regard to anthropogenic global 
warming (AGW). In a sense, it seems to be a conclusion in search of a method, as S&O note it has 


been used to “quantify the consensus on AGW” (p. 2). 


In the S&O research, consensus measurement is not given full explanation; their cited source, Cook 
et al. (2013), present consensus measurement as an examination of past research (as in a meta- 
analysis, but lacking the statistical rigor), while allowing comparison and change, with any 
disagreements resolved by a third party. Further, S&O state that in their study they “adapt and 
combine the methodologies” of consensus measurement and content analysis (p. 2) without 
specifying how this was done. S&O have not applied consensus measurement in the usual way, that 
is, by estimating a numerical “consensus rate.” Additional information is needed to fully assess how 
these decisions affected the coding scheme and other aspects of the study, but, at a minimum, the 
blended methodology is unreliable and the inherent biases of consensus measurement were 


introduced into the study. 


5 Meta-analysis is a statistical procedure for combining data or results from multiple studies on the same phenomenon, 
increasing statistical power and creating a total estimated effect and allowing an overall conclusion. 
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VI. B. Consensus measurement is practiced by a limited circle of researchers. As referred to in the S&O piece, 
consensus measurement seems to be located within the purview of a specific group of researchers. 
The investigators using consensus measurement seem to be a relatively small group, with inter- 
citation and self-citation notable (e.g., Anderegg & Goldsmith, 2014; Cook, 2016; Cook & Jacobs, 
2014; Cook et al., 2013; Cook et al., 2016; Maibach & van der Linden, 2016; Oreskes, 2004). This 
type of interdependence has the potential to create an “echo chamber” of reinforcing ideas, without 


critique and correction (see, e.g., Jankó, Vancsó, & Móricz, 2017). 


VIL C. Consensus measurement has been criticized in the academic literature. Critiques of consensus 
measurement as practiced by S&O have been produced by independent critics (e.g., Stirling, 2017), 
and in peer-reviewed form (Pearce et al., 2017), including in the journal that published the S&O 
research, Environmental Research Letters (e.g., Dean, 2015; Tol, 2016). In particular, Tol (2016) 
conducted a thorough analysis of the “highly influential” Cook et al. (2013) study of consensus 
within the scientific literature concerning AGW. Tol identifies a number of limitations of the 
techniques used by Cook et al., including a failure to take into account systematic interrater 
differences, possible non-independence of raters, and discrepancies with regard to how the 
documents sampled were collected. Tol notes that when papers or experts that do not take a 
position on the human impact on global warming are included in the analysis, the consensus rate 
drops from 96%—98% to 33%—63%. These factors preclude the technique’s reliability, objectivity, 
and validity. 


CONCLUSION 


The above analysis documents the numerous fundamental and fatal flaws in the study’s content 
analysis. In short, the content analysis is unreliable, invalid, biased, not generalizable, and not 
replicable. Accordingly, S&O provide no scientific support for either a discrepancy among 


ExxonMobil’s climate change communications, or a claim that ExxonMobil misled the public. 
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